Augmented reality apparatus and method

ABSTRACT

A method of providing an augmented reality image comprises recording a basic image including a subject and a first background using a recording device, extracting a subject image from the basic image, and providing the extracted subject image to a display device for combining with a second background. At A the sending person using the app to record a moving image of their own head—ie a video—which is separated from the background by the app (the background being automatically discarded). The image is then sent to a recipient who, at B, sees the head speak to them either on their desktop or in the camera view of the smart phone/tablet if they so choose.

PRIORITY

The present application is a continuation of U.S. National Stage Entryapplication Ser. No. 15/317,011, filed Dec. 7, 2016, which claimspriority to International Patent Application No. PCT/GB2015/051697,filed Jun. 10, 2015, which claims priority to United Kingdom (UK) PatentApplication No. GB1410285.9, filed Jun. 10, 2014; UK Patent ApplicationNo. GB1413146.0, filed Jul. 24, 2014; and UK Patent Application No.GB1502854.1, filed Feb. 20, 2015, the disclosures of which are hereinincorporated by reference in their entireties.

The present invention relates to an apparatus and a method for providingan augmented reality experience.

Augmented reality, in which the viewing of a real world environment isenhanced using computer generated input, is becoming available onvarious platforms, including television, head up displays, and to alimited extent, hand held devices such as cell phones.

The use of hand held devices, such as cell phones, as cameras has beenenhanced by the availability of small, specialised downloadableprograms, known informally as apps. Many of these include computergenerated visual effects that can be combined with a “live view” throughthe camera, to provide the user with a degree of augmented reality foran improved image or amusement. However the incorporation of videofootage into the live view of a camera has proved to be difficult due tothe limited processing power available in most hand held devices, andthe lack of a functional codebase provided with the built-in frameworks.

Our published UK patent application, GB 2 508 070 describes examples oftechniques for generating effective augmented reality experiences onhand-held devices.

Embodiments of the present invention aim to provide apparatus and amethod for capturing an image of a subject on a camera device,extracting the image from a background and sharing the extracted image.

Embodiments of the present invention also aim to provide a system andmethod for sharing images to be used in augmented reality displays.

The present invention is defined in the attached independent claims towhich reference should now be made. Further, preferred features may befound in the sub-claims appended thereto.

According to one aspect of the present invention there is provided amethod of providing an augmented reality image, the method comprisingrecording a basic image comprising a subject and a first backgroundusing a recording device, extracting a subject image from the basicimage, and providing the extracted subject image to a display device forcombining with a second background.

The second background may comprise any of, but not limited to: a desktopbackground, e.g. a display screen of a device, a background provided byan application or a background captured by a camera. The background maybe captured by a camera of a device on which the subject image is to beviewed.

Preferably the extracted subject image is provided to the display devicefor combining with a second background as imaged by a camera of thedisplay device.

In one embodiment the recording device and the display device are partsof a common device, which may be a hand-held device. Alternatively or inaddition the recording device and the display device may be separate andmay be located remotely. The recording device and display device mayeach be part of separate devices, which devices may be hand-held devicesand which devices may comprise, but are not limited to, mobiletelephones and tablets.

The recording and display devices may comprise different types ofdevice.

In a preferred arrangement the first and second backgrounds aretemporally and/or spatially separate. The first background may comprisean image that is contemporaneous with the subject image and the secondbackground may comprise an image that is not contemporaneous with thesubject image.

In a preferred arrangement the step of extracting the subject from thebasic image is performed locally with respect to the recording device,and preferably within the device. Alternatively, the step of extractingthe subject image from the basic image may be performed remotely fromthe recording device.

The step of extracting the subject image from the basic image may beperformed in real time, with respect to the recording of the basicimage, or else may be performed after recording of the basic image.

Preferably the method comprises sending the extracted subject image fromone device to another device. The image is preferably a moving image,and more preferably a moving, real-world image.

The extracted subject image may comprise a head and/or face of a user,such as of a sender of the image. The image is more preferably a movingimage and may include, be attached to, or be associated with, an audiofile, such as a sound recording of, or belonging to, the moving image.

The image may include one or more graphical elements, for example anaugmented reality image component. The augmented reality image componentmay be anchored to the extracted subject image so as to give theappearance of being a real or original element of the extracted subjectimage.

In a preferred arrangement the method includes sending an extractedsubject image, preferably a moving image, over a network to a recipientfor viewing in a recipient device. Optionally a sound recording may besent with the extracted subject image. Alternatively, or additionally,the method may include sending the extracted subject image directly to arecipient device.

In a preferred arrangement, the method comprises recording a basic imagecomprising a subject and a first background, extracting a subject fromthe background as a subject image, sending the subject image to a remotedevice and combining the subject image with a second background at theremote device.

The method may include extracting a subject from a basic image by usingone or more of the following processes: subject feature detection,subject colour modelling and subject shape detection.

According to another aspect of the present invention there is providedapparatus for providing an augmented reality image, the apparatuscomprising a camera for recording a basic image comprising a subject anda first background using a recording device, an image processor forextracting a subject image from the basic image, and a display devicefor combining the extracted subject image with a second background.

Preferably the extracted subject image is arranged in use to be combinedwith the second background as imaged by a camera of the display device.

In one embodiment the recording device and the display device are partsof a common device, which may be a hand-held device. Alternatively or inaddition the recording device and the display device may be separate andmay be located remotely. The recording device and display device mayeach be part of separate devices, of which one or both may be a handheld device.

In a preferred arrangement the first and second backgrounds aretemporally and/or spatially separate. The first background may comprisean image that is contemporaneous with the subject image and the secondbackground may comprise an image that is not contemporaneous with thesubject image.

The processor may be arranged in use to extract the subject from thebasic image locally with respect to the recording device, and preferablywithin the device. Alternatively, the processor may be arranged in useto extract the subject image from the basic image remotely from therecording device.

The processor may be arranged in use to extract the subject image fromthe basic image in real time, with respect to the recording of the basicimage. Alternatively, the processor may be arranged in use to performthe extraction after the recording of the basic image.

The subject image may comprise one that has been previously stored.

The subject image may comprise a sequence of still images taken from amoving video.

Alternatively or additionally the subject image may comprise acontinuous moving video image.

The subject image may comprise an image of a person or creature, orcould be any other “real world” object or item.

For viewing the image, a context identification unit may be arranged inuse to identify a context for the subject image. This may be achieved bycomparing at least one object in a field of view with stored data from aplurality of objects. An image retrieval unit may be arranged to selectan image from a plurality of stored images according to contextinformation determined by the context identification unit. A positioningunit may be arranged in use to position the subject image in abackground. This may be achieved according to context informationdetermined by the context identification unit.

The positioning of the subject image by the positioning unit may includesizing of the subject image in the display, and may include anchoringthe subject image in the display, preferably with respect to contextinformation determined by the context identification unit.

The context identification unit, and/or the retrieval unit, and/or thepositioning unit may comprise processes arranged in use to be performedby one or more electronic processing devices.

The invention also provides a program for causing a device to perform amethod of providing an augmented reality image, the method comprisingrecording a basic image comprising a subject and a first backgroundusing a recording device, extracting a subject image from the basicimage, and providing the extracted subject image to a display device forcombining with a second background.

The program may be contained within an app. The app may also containdata, such as subject image data and/or background image data.

The invention also provides a computer program product, storing,carrying or transmitting thereon or therethrough a program for causing adevice to perform a method of providing an augmented reality image, themethod comprising recording a basic image comprising a subject and afirst background using a recording device, extracting a subject imagefrom the basic image, and providing the extracted subject image to adisplay device for combining with a second background.

According to another aspect, there is provided a messaging system,comprising a plurality of devices wherein at least a first, sending userdevice is arranged in use to send an image to at least a second,receiving user device, which image comprises an electronically capturedimage, for displaying as a virtual image in an augmented reality on adisplay of the second user device.

The devices may be arranged to communicate directly, and/or via acommunications network including a processor-based server.

In a preferred arrangement, at least the receiving user device comprisesa context identification unit for identifying a context from at leastone real image captured by the device, and a virtual image positioningunit for positioning the virtual image in the display.

Preferably the virtual image is one that has been previously stored.

In a preferred arrangement the virtual image comprises a sequence ofstill images taken from a moving video.

Alternatively or additionally the virtual image may comprise acontinuous moving video image.

The virtual image may comprise an image of a person or creature, orcould be any other “real world” object or item.

In a preferred arrangement the context identification unit is arrangedin use to identify a context by comparing at least one object in a fieldof view with stored data from a plurality of objects. The imageretrieval unit is preferably arranged to select an image from aplurality of stored images according to context information determinedby the context identification unit. The positioning unit is preferablyarranged in use to position the virtual image according to contextinformation determined by the context identification unit.

The positioning of the image by the positioning unit may include sizingof the image in the display, and may include anchoring the image in thedisplay, with respect to context information determined by the contextidentification unit.

The context identification unit, and/or the virtual image retrievalunit, and/or the virtual image positioning unit may comprise processesarranged in use to be performed by one or more electronic processingdevices.

The virtual image may comprise an extracted subject image according toany statement herein.

The invention also includes a method of sending an image from one deviceto another device wherein at least a first, sending user device sends animage to at least a second, receiving user device, which image comprisesan electronically captured image, and the image is displayed as avirtual image in an augmented reality on a display of the second userdevice.

The image may be sent via a communications network including aprocessor-based server.

The image may be according to any statement herein.

The invention may include any combination of the features or limitationsdescribed herein, except such a combination of features as are mutuallyexclusive.

Preferred embodiments of the present invention will now be described byway of example only with reference to the accompanying diagrammaticdrawings in which:

FIG. 1 shows a virtual image superimposed upon a camera view of a realimage, in accordance with an embodiment of the present invention;

FIG. 2 shows schematically a first step in a context recognition processin accordance with an embodiment of the present invention;

FIGS. 3 and 3 a show schematically an alternative first step in acontext recognition process, in which there are multiple visible objectsin the camera live view;

FIG. 4 shows schematically an animation technique for use with anembodiment of the present invention;

FIG. 5 shows schematically a positioning process according to anembodiment of the present invention;

FIG. 6 shows schematically optional user controls for a virtual image,according to an embodiment of the present invention;

FIG. 7 shows a first step in an anchoring process for the image of FIG.6;

FIG. 8 shows a further step in the anchoring process of FIG. 7;

FIG. 9 shows schematically an alternative anchoring process according toan embodiment of the present invention;

FIG. 10 shows schematically an automatic re-sizing process for a virtualimage, in accordance with an embodiment of the present invention;

FIG. 11 shows schematically an automatic re-sizing process for a virtualimage in an alternative scenario;

FIGS. 12-15 show schematically different steps in a process for taking aphotograph incorporating both a real and a virtual image, according toan embodiment of the present invention;

FIG. 16 shows schematically a process for acquiring video footageincorporating both real and virtual images;

FIG. 17 is a schematic flow diagram showing some key steps in theprocess of displaying a virtual image in the live view of a camera, inaccordance with an embodiment of the present invention;

FIGS. 18-23 show a method for extracting a subject image from abackground and sending the extracted image to a remote device;

FIGS. 24-31 show a messaging platform according to an embodiment of thepresent invention;

FIGS. 32 and 33 shows a number of extracted subject images displayed onscreens of recipient devices, in accordance with an embodiment of thepresent invention;

FIG. 34 shows a further embodiment of messaging according to the presentinvention, and

FIG. 35 shows schematically some of the processes used in extraction ofa subject image from a basic image including the subject and abackground.

The embodiment described below aims to provide an augmented realityexperience to a user of a hand held device, such as a mobile phone,which incorporates an electronic processor, a camera and a display. Inparticular, images taken from video footage are displayed in a displayof a hand held device together with a live camera view, to create theillusion that the subject of the video—ie the virtual moving image—ispresent in the field of view of the camera in real time.

In this context the term “real world” image means an image taken fromreality, such as a physical, real-world scenario using an electronicphoto-capture technique, e.g. video recording.

In order to achieve this the device must undertake various processes,including acquiring contextual information from the camera view,obtaining an appropriate virtual image, positioning the virtual imagewithin the camera view, optionally anchoring the virtual image withrespect to the context and optionally sizing the virtual image withinthe camera view.

The processes may be performed by an electronic processor of the handheld device.

The data necessary for the reconstruction of the virtual moving image,together with one or more programs for facilitating the necessaryprocesses for manipulating it to provide the augmented realityexperience, are downloadable to a hand held device in the form of aspecialist program, or software application, known widely as an app. Theapp can preferably be updated to present the user with fresh viewingexperiences.

The first example described in detail below is that of an augmentedreality system for use as a guide at a visitor attraction, in which avirtual image of a figure is displayed within the real world camera viewto provide information, via an associated audio file, about theattraction.

Turning to FIG. 1, this shows schematically a camera 10 of a hand helddevice, in this case aimed at a well-known object 12, which isrecognisable to the device, and a moving virtual image 14 of an actorplaying the part of an historical figure that is chosen because of itsrelevance to the object 12. The device recognises the object, in thiscase a statue, based upon a unique set of matrix points 12 a which havebeen stored in the downloaded app in an earlier stage, and which canprovide the device with contextual information necessary for thesubsequent selection, display and manipulation of the virtual image 14.

Moving virtual images 14 are stored in the device as sequences of stillimages taken from a video file, and synchronised with an appropriateaudio file, when the app is downloaded and the appropriate one is chosenafter the context has been determined.

Turning to FIG. 2, this shows the chosen virtual image 14 as it isdisplayed in the camera view of the device, beside the object 12.

FIG. 3 shows schematically the scenario in which multiple objects aredetected by the device. In this case the object 12 is detected and soare two further objects 16 and 18. The device displays all three objectstogether with respective virtual buttons superimposed thereon so thatthe user may select the object of interest by touching the appropriatebutton on the screen, as is shown in FIG. 3 a.

FIG. 4 shows schematically one method for animating a virtual image. Ituses a long established technique of cutting a moving image into asuccession of still frames 20 on a green screen background (not shown).The device then plays back the sequence of still images, removing thegreen screen background automatically as necessary. As the individualimages are replaced at a rate greater than six frames per second, thehuman eye interprets them as a continuous moving image. A soundtrack,optionally of MP3 format, is played in synchronism with the animation toreinforce the illusion of continuous video footage. In this example theanimated figure is a Roman soldier, whose commentary and actions arerelevant to the attraction being viewed through the camera display.

FIG. 5 shows schematically a technique for positioning the image 14 withrespect to the object 12. During creation of a particular app, when theparticular scene is first investigated, a creative director will choosean optimum placement for the virtual image, based upon a number offactors, both artistic and practical. Once the optimum position ischosen the system uses trigonometry to compute the position of the imageat real world spatial coordinates x, y and z with respect to the object12. An alternative is to decide upon a zero point within the object andto position the image using absolute x, y and/or z coordinates from thezero point.

FIG. 6 shows schematically how the user can re-size or reposition theimage with respect to the object. The image can be resized using afinger and thumb pinching and spreading technique 22 whilst touching thescreen. The image can be moved using a drag and drop technique 24, andan anchoring system (described below) can also be activated ordeactivated by a double finger double tap technique 26. In a PLAY mode,a virtual PAUSE button 28 is also provided, which converts to a virtualPLAY button (not shown) in PAUSE mode.

In order to maintain the illusion that the figure is actually presentbeside the attraction, it is necessary that the position of thefigure—ie the image 14—be spatially anchored with respect to the object12. This is because if the user moves whilst viewing the object and thevirtual image through the camera, an image that is fixed with respect tothe camera screen would quickly fail to maintain the illusion ofreality.

FIG. 7 shows schematically an anchoring system according to oneembodiment of the present invention. The system uses a pre-definedalgorithm to seek objects that are either prominent or else have adefinitive shape within the camera view. Once several objects have beenlocated the system uses advanced trigonometric techniques to evaluatethe scene displayed in the camera view and to allocate proportion datato the virtual image. The system then locks the image in x, y and zcoordinates with respect to its real world context.

FIG. 8 shows schematically in more detail the anchoring system accordingto the above-described embodiment of the present invention. Firstly, alabel 30 indicates that the anchor system has been activated. Then thedevice dynamically detects the nearest object 32 in the camera view. Inthis case, the method used is one in which an algorithm seeks torecognise objects by detecting a pattern, rather than usingpre-processed matrix points (as per the example of FIG. 1). This allowsthe algorithm to look for real world objects to which the performance—iethe virtual image—can be anchored. For example, the algorithm couldrecognise the four edges of a snooker table. This allows an improvedanchoring technique as recognition rules are created that allow theapplication of higher or lower thresholds based upon a particularobject, or type of object. One suitable previously considered algorithmis known as FAST (Features from Accelerated Segment Test).

A second object 34 is then detected by the device, to provide depthinformation. The image is then anchored to the first object—ie theposition of the image in x, y and z coordinates with respect to thelocation of the first object 32 is determined. The device then checksregularly to determine whether the object pattern—ie of objects 32 and34—has changed, which would occur if the user holding the device hadmoved. If the device determines that there has been movement the devicere-scans the field of view and determines the closest match to theinitial pattern of objects 32 and 34 to ensure that the position of thevirtual image 14 is still true.

The above-described approach allows a user to lock the anchor to a knownobject within the display, in almost any location, efficiently andinvisibly. If there is no specific object from which to take areference—such as an open field, for example, then the system revertsfirstly to a pre-loaded recognition library and then if no view isrecognised a digital compass and GPS reference are used to fix thelocation of the image in real space.

The use of GPS and digital compass bearing by the anchoring system isdepicted schematically in FIG. 9. This configuration builds a basic realworld map by using GPS coordinates alongside compass bearings. The GPScoordinates are used to lock a known longitude and latitudeconfiguration, whilst the bearings are used to detect 360 degreecircular movement by the user. If the system detects such a movementthen the movie is returned to its original locked position. Theanimation returns using algorithms that provide a smooth and quickreturn to the coordinates acknowledging dampening and speed of returnbased on distance moved.

The apparent size of the image with respect to objects in the cameraview is also important to maintain the illusion of reality. FIG. 10shows an automatic sizing operation in which the image 14 is adjustedwith respect to the object 12 when a user, viewing the object throughthe camera device, moves either closer to or further away from theobject.

Sophisticated algorithms are employed by the device to adjust the sizeof the image smoothly as the user moves towards or away from the object12. The autofocus function of the camera lens may be employed to providedata concerning a change in the distance from the object. If the devicedoes not possess an autofocus function then the distance to therecognised object can be calculated using stored data about its origin.Both techniques can be used, where available, to provide a more accuratereading.

Alternatively, the user can manually re-size the image 14 using thepinch technique 22 described earlier.

If a relatively cramped location is detected by the system, such as anindoor location, or a medieval street scene for example, the deviceautomatically re-sizes the image to a larger size so as to maintainrealism. FIG. 11 depicts the enlarged image 14 in such a case.

The system also allows the capture of still or video images bearing boththe real view and the virtual image. FIGS. 12-15 show schematically aprocess for taking a photograph with the virtual image 14 included. InFIG. 12 a real person 36 walks into a scene in which the virtual image14 of the figure is already positioned. In FIG. 13 the photograph istaken and stored and the coordinates of the virtual image are recorded.In FIG. 14 the system post-processes the image 14 and the perspective isauto-detected. The composite image is then re-saved. At FIG. 15 the useris invited to share the stored composite image via virtual buttons 38accessing several common media. A short video sequence can be recordedand shared in a similar way.

FIG. 16 shows schematically an example of a process according to theembodiment described above.

At step 100 the process begins. At step 102 object recognition rules areread from a database. At step 104 the device reads the view and at step106 it checks for a recognisable pattern. The device loops until apattern is detected. Once a pattern is detected an appropriate movingimage is selected from a library at step 108. At step 110 the image ispositioned and play begins. Step 112 awaits a user input. Options toexit 114, re-size 116, anchor 118 or reposition 120 are available. Ifthe user selects to exit the app at step 114 the app is stopped at step122. Otherwise the video image continues to play at step 124.

FIG. 17 shows an alternative embodiment in which an object recognitiondatabase 40 is split into several smaller databases 42, 44, 46, 48according to user location. Three or more angles of an object arechecked 50 and once the object has been detected the virtual image islaunched 52.

The above examples describe using touch controls, which may be differentto the ones described. However, where the apparatus supports it,non-contact gestures may be employed to control the device. Similarly,where the apparatus supports it, voice commands may be used to controlthe apparatus.

The term “context”, as used herein, may in most cases be regarded as acamera view. Likewise, the term “contextual information” may in mostcases be regarded as “camera view information”.

The contextual information may be derived from a “real world” image, asviewed through the camera of the device, or may be derived from atwo-dimensional image, such as a printed page, photograph orelectronically displayed image. This allows the techniques describedabove to be used to enhance a user experience in a wide variety ofcircumstances, such as viewing a printed publication or advertisement.In one embodiment (not shown), the virtual image can be made to appearto rise or “pop” up from a such a two dimensional context.

Image processing techniques may be employed to create virtual shadowsfor the virtual image, so as to enhance the perception that the virtualimage is a real one. Similarly, image processing techniques may beemployed to balance the apparent brightness of the virtual imagerelative to the real world context being viewed through the device.

Although the examples described above are of a two-dimensional viewingexperience, the techniques described herein may also be applied to anapparent three-dimensional viewing experience where the apparatussupports this, such as in 3-D video playback formats.

In the above description, the term “virtual image” is intended to referto a previously captured or separately acquired image—which ispreferably a moving image—that is displayed on a display of the devicewhilst the user views the real, or current, image or images beingcaptured by the camera of the device. The virtual image is itself a realone, from a different reality, that is effectively cut out from thatother reality and transplanted into another one—the one that the viewersees in the display of his device.

Turning to FIGS. 18-23, these show a method of recording a basic imageof a subject and a background, extracting the image of the subject fromthe background and sending it to a remote device.

The method comprises three basic steps, as outlined in FIG. 18:

-   -   1. Shoot a video clip.    -   2. Select and send.    -   3. Appear in the recipient's environment.

In more detail (FIG. 19):

The user takes a video sequence of a subject. The video sequence ismonitored for optimum duration.

A premium version of the invention may permit a longer video sequence.

The user will decide to keep or delete the video sequence. If accepted,the methodology moves forward.

-   -   Edit the First Frame (FIG. 20):

Once the user is happy with the video sequence they are presented withthe first frame.

This first frame will be used to identify the subject.

Once the subject has been identified it will be tracked through thesystem frame-to-frame.

The user selects the subject by either of the following:

The whole image will be grayscale and the user will use a finger tohighlight the object by replacing the missing colour values. This methodforms boundary edges.

Or

The user will select around the object forming a ‘halo’ to define theboundary edges.

For immediate purpose the first of the above technique will beexplained.

Select the Object (FIG. 21):

The user uses a finger to select the subject.

The app/method will allow for different size area selectors, as well asa zoom feature for minute accuracy.

The app/method will also allow for a solid colour overlay to show anyareas of the subject that have been missed by the user.

Once the user is happy with the subject selected, they will then passthe object to the next stage of the methodology.

Realise the Background (FIG. 22):

The subject selected, now allows for inverse recognition of thebackground.

The background will be selected and deleted from the subject.

The background will be filled with a solid defined colour, or shaded orotherwise differentiated from the subject.

Note that the distinction between the subject and the background neednot be a simplistic, binary background/subject label. Instead each pixelcould be labelled with a probability that it belongs in the subject, andthe determination could be made according to the probability value.

Track the Object (FIG. 23):

The initial frame background has now been deleted.

The system will now analyse each frame:

1. Follow the motion of the subject.

2. Delete the background information.

3. Replace background with solid colour or otherwise contrastingindicator.

Once each frame has been rendered they can be ‘stitched’ back togetherinto a video sequence, or returned as a set of frames.

This may be completed on the mobile device, or may be completedoff-device on an external computing system where it would be returned tothe user's device once ready.

The user will decide whether the image is as they require. If they wishto keep it will be stored both on the device alongside a version withinan off-site hub/database.

This rendered image can then be sent to any other user who has the apptechnology installed on the device.

There are several methods by which the subject image could be extractedfrom the basic image including the background. The actual method usedmay comprise one or a combination of the following methods.

Method 1. Simple Background.

In this method, a user/subject stands in front of a solid colour wall,with enough space in front of the wall to avoid shadows. The user needsto wear contrasting colour clothes that stand out from the backgroundwith soft lighting.

Extraction of background is then based on the colour at the edge of theimage (top/sides).

If the converted video is to be played back on a mobile device thebackground alpha can be faded so that there are no sharp edges at thebackground/foreground image border. In this case the alpha will not beencoded as a simple green screen but will use an extra scale image ineach frame of the video to encode the alpha channel.

This extraction method is relatively straightforward to implement.However, to get the best result will need some experimentation and couldhave some user input to guide the finished result. For example drawing acrude outline of the desired part of a video frame to include/excludewould give a greater range of colours that would be defined asbackground.

The conversion time could be fast, at or near real time.

Method 2, Background Static.

Background subtraction is a major pre-processing step in manyvision-based applications. OpenCV has implemented three such algorithms.The basic idea is to assume the background is static and identifychanges in the image. These methods will also extract shadows asforeground and may not detect large solid block of colour as foreground.Camera movement would be a problem as this would appear as newforeground at the edge of images. Camera movement can be restricted bythe users and video stabilisation could be used where present in themobile device.

It is also possible to add some user input to the background detectionto identify regions that are known background/foreground.

This method can be combined with Method 1 to give a better result thanMethod 1 alone as it would allow for backgrounds that were more variedthan a blank wall.

This method may require a processing stage after the video was taken.

Method 3 (Interactive with User Input—Using the Example of OpenCV)

Interactive Foreground Extraction Using GrabCut Algorithm (OpenCV)

OpenCV has an algorithm for extracting the foreground subject that givesvery good results, the GrabCut algorithm. GrabCut requires the user toprovide information at the region of interest and/or provide hints tothe probable foreground/background. In the first pass the extraction isdone and results presented. The user must then mark the areas that arewanted but have been removed or been removed but are wanted, as probableforeground/background.

For video the user defined information can be carried forward to thenext frame. If required the region of interest and probableforeground/background could be updated. An intelligent implementationwould attempt to follow the probable foreground/background to automatethe process as far as possible.

Changes to the original algorithm would allow the background/foregroundalpha to be based on the probability of each allowing a soft edge to theselection.

This method may not be real time and may require a processing stageafter the video was taken, but can achieve high quality results.

A fourth method—based around a specific type of subject e.g. a face—willbe described later.

Implementation

Turning to FIGS. 24-30, these show schematically a chat system ormessaging platform according to an embodiment of the present invention,in which a mobile phone is the example of hand held device on which theinvention is carried out.

Firstly the phone is tilted to near vertical and the camera opens with afrosted effect (FIG. 24).

The user types messages or presses an app button to open image packs.The images may be pre-recorded images of the user, for example, or maycomprise images downloaded from a commercial site.

The next step (FIG. 25) is to select the pack from a collection.

The user then touches the chosen image and it launches on the screen(FIG. 26). The image is then tapped again to send.

FIG. 27 shows the next stage. The recipient sees the sent image on theirscreen. They can tap to obtain/view the pack.

The user is able to create his/her own personalised images FIG. 28). Forexample the user is able to show off a new dress, invite someone to aparty or simply show a series of emotions.

A video clip can be made (FIG. 29) which can then be selected and sent,as described above, so that it appears in a recipient's environment(FIG. 30).

FIG. 31 shows the sequence of steps schematically. At Step A, a person,in this case a child, is recorded as a moving image. The background tothe image is then separated and discarded before the selected image issent to a recipient who can be anywhere in the world. Image B depicts areal scene at a remote location. In this case the scene is of a child'sparty. Having received the moving image of the girl, the recipient canthen play back the received image superimposed onto the camera viewdisplayed on the recipient's screen—as depicted at C. A video can thenbe made of the virtual encounter whereby a person at a remote locationcan appear to be present in the recipient's world, with the virtualimage appearing to be anchored in the recipient's environment, so as tocreate the illusion that the subject depicted in the image is actuallypresent in the recipients space.

Turning to FIG. 32, this shows various images 100 on displays 110 asreceived by recipients. The method and apparatus described above allowsomething between a text message exchange and a video call. The messagesender uses either the front camera or the rear camera in the smartphone to capture a short video of them speaking and the app softwarecuts out the sender's head 100 before transmitting the video clip toappear on the recipient's screen 110. The cut-out head can appear on arecipient's desktop, conveniently as part of a messaging screen.Alternatively the recipient, who also has the app, can open therear-facing camera of their phone so that the head appears to float intheir environment 112 as it plays the short performance. The process isreversed to make a reply.

FIG. 33 shows the process schematically. At A the sending person usingthe app to record a moving image of their own head—ie a video—which isseparated from the background by the app. In a preferred arrangement thebackground can be automatically discarded. However the person who makesthe recording could instead manually remove the background as analternative, or additional, feature. The image is then sent to arecipient who, at B, sees the head speak to them either on their desktopor in the camera view of the smart phone/tablet if they so choose.

Such a message according to the embodiment is different to a textmessage because:

-   -   It is faster to use than tapping out character keys    -   It conveys emotion as the facial expression can be seen and tone        of voice heard, rather than just words on a screen    -   It can be both funny and personal    -   The users can take/store photos of the head, if the sender        grants permission.

The message is different to a video call because:

-   -   It uses very small amounts of the mobile user's data allowance.    -   It delivers discreet, individual ‘sound-bites’ of message    -   It has the option to add on augmented reality images, locked to        the head, such as those shown at 114, including horns, a hat and        stars, in the examples shown.    -   It can easily be kept for future reference

With embodiments of the present invention as described above, images,including moving or video images, can be sent by a sender to a receiverto appear in the receiver's environment as a virtual image when viewedthrough a display of the receiver's device, against a receiver'sbackground being imaged by a camera of the receiver's device. The imagecan be locked or anchored with respect to the background being viewed,so as to give the appearance of reality.

The images can comprise images created by the sender and extracted as asubject from a sender's background, to be viewed against a receiver'sbackground. Furthermore, the images can be sent from user to user over aconvenient messaging network.

It should be noted that with the methods described above, the sender isable to send an image of himself without revealing hisbackground/whereabouts to the recipient.

The foreground, or subject, image can be sent without the background,and not merely with the background being made invisible (e.g. alphavalue zeroed) but still remaining part of the image.

Also, the examples above have the recipient viewing the received imagethrough the camera view of the recipient's device, this need not be thecase. For example, as an alternative the recipient may view the imagefloating on his desktop or above an app skin on his device. This may bemore convenient to the user, depending on his location when viewing.

Where the image to be sent comprises e.g. just a head of the sender,this represents a relatively small amount of data and so the inventioncan provide a systemised approach to sending video images without theneed for the usual steps of recording a video clip, saving, editing andthen sending it to a recipient.

FIG. 34 shows a sequence of steps (from left to right) in a messagingprocess, in which a combination of the abovementioned options may bebrought into the user experience. A hand held device 200 is used toconvey messages in the form of speech bubbles between correspondent Aand correspondent B according to a known presentation. However,correspondent A also chooses to send to correspondent B a moving image210 of her own face, delivering a message.

In this example the conversation raises the subject of a performance bya musical artiste. One of the correspondents A and B can choose to sendto the other an image 220 of the artiste's head, which then appears onthe desktop. The moving image can also speak a short introductorymessage. This is available via the messaging app being run by thecorrespondents on their respective devices. If the head 220 is tappedwith a finger 230, a fuller image 240 of the performer appears on top ofthe graphical features seen on the desktop to deliver a song, or otherperformance.

If the full image 240 is tapped by the finger 230 again it opens thecamera (not shown) of the device so that a complete image 250 of theperformer is integrated with a background image 260 of user'senvironment, in scale and anchored to a location within the backgroundimage so that it remains stationary with respect to the background ifthe camera move left/right or in/out to give the illusion of reality.

Thus, using the various aspects and/or embodiments of the inventiondescribed above, a user can switch between a cut-out part, such as ahead, of a selected moving image, a fuller image and a completeaugmented reality experience. Moreover this facility can be employed ina messaging system, between two or more correspondents.

The above described techniques can be used in other platforms, such asdirect or peer-to-peer messaging platforms, in which a network need notbe required. They can also be used for business, such as in businessconferences, as well as for purely social interaction.

The above-described embodiments may also be used as part of a videovoicemail system.

Furthermore, whilst in the above-described examples the userscommunicate using hand held devices such as mobile phones and/or tabletcomputers, the devices used need not be of the same type for both senderand receiver or for both/all correspondents in a messaging system. Thetype of device used may be any of a wide variety that has—or can connectto—a display. Gaming consoles or other gaming devices are examples ofapparatus that may be used with one or more aspects of the presentinvention.

The process of extracting a subject from an image including an unwantedbackground is sometimes referred to as “segmentation”. The followingdescription is of techniques for performing segmentation when thesubject belongs to a known class of objects.

Method 4: Face Segmentation

When the source video comprises an object taken from a known objectclass, then object-specific methods for segmentation can be employed. Inthe following example human faces are to be segmented where the video isa spoken segment captured with the front facing camera (i.e. a “videoselfie”). The same approach could be taken with any object class forwhich class-specific feature detectors can be built.

The face-specific pipeline comprises a number of process steps. Therelationship between these steps is shown generally at 300 in theflowchart in FIG. 35. In order to improve the computational efficiencyof the process, some of these steps need not be applied to every frame F(instead they are applied to every nth frame) of the input videosequence IS. A detailed description of each step is as follows:

In Process 310 facial feature detection is performed. The approximatelocation of the face and its internal features can be located using afeature detector trained to locate face features. Haar-like features aredigital image features used in object recognition. For example, acascade of Haar-like features can be used to compute a bounding boxaround the face. Then, within the face region the same strategy can beused to locate features such as the eye centres, nose tip and mouthcentre.

In Process 320 skin colour modelling is performed. A parametric model isused to represent the range of likely skin colours for the face beinganalysed. The parameters are updated every nth frame in order to accountfor changing appearance due to pose and illumination changes. In thesimplest implementation, the parameters can be simply the colour valueobtained at locations fixed relative to the face features along with athreshold parameter. Observed colours within the threshold distance ofthe sampled colours are considered skin like.

A more complex approach is to fit a statistical model to a sample ofskin pixels. For example, using the face feature locations, a set ofpixels are selected that are likely to be within the face. Afterremoving outliers, a normal distribution is fitted by computing the meanand variance of the sample. The probability of any colour lying withinthe skin colour distribution can then be evaluated.

In order to reduce the influence of colour variations caused by lightingeffects, the model can be constructed in a colour space such as HSV orLCrCb. Using the H channel or the Cr and Cb channels, the model capturesthe underlying colour of the skin as opposed to its brightness. AtProcess 330 shape features are determined. The skin colour modelprovides per-pixel classifications. Taken alone, these provide a noisysegmentation that is likely to include background regions or missregions in the face. There are a number of shape features that can beused in combination with the skin colour classification. In the simplestimplementation, a face template such as an oval is transformed accordingto the facial feature locations and only pixels within the template areconsidered. A slightly more sophisticated approach uses distance tofeatures as a measure of face likelihood with larger distances beingless likely to be part of the face (and hence requiring more confidencein the colour classification).

A more complex approach also considers edge features within the image.For example, an Active Shape Model could be fitted to the featurelocations and edge features within the image. Alternatively, superpixelscan be computed for the image. Superpixel boundaries naturally alignwith edges in the image. Hence, by performing classifications on eachsuper-pixel as opposed to each pixel, we incorporate edge informationinto the classification. Moreover, since skin colour and shapeclassifiers can be aggregated within a superpixel, we improverobustness.

At process 340 Segmentation takes place. Finally, the outputsegmentation mask OM is computed. This labels each pixel with either abinary face/background label or an alpha mask encoding confidence thatthe pixel belongs to the face. The labelling combines the result of theskin colour classification and the shape features. In the implementationusing superpixels, the labelling is done per-superpixel. This is done bysumming the per-pixel labels within a superpixel and testing whether thesum is above a threshold.

Whilst endeavouring in the foregoing specification to draw attention tothose features of the invention believed to be of particular importance,it should be understood that the applicant claims protection in respectof any patentable feature or combination of features referred to herein,and/or shown in the drawings, whether or not particular emphasis hasbeen placed thereon.

The invention claimed is:
 1. A method of messaging between a senderdevice and a recipient device including a recipient display, the methodcomprising: recording a video of a live camera view using a camera inthe sender device, the video of the live camera view including at leasta face of a person and a first background; extracting a moving image ofat least a face of the person from the video of the live camera view atthe sender device to create a video sequence, the extracted moving imageof at least the face of the person excluding images of the firstbackground from the live camera view; sending the video sequence of theextracted moving image of at least the face of the person as a messageto the recipient device without the first background; superimposing thevideo sequence of the extracted moving image of at least the face of theperson on a second background of the recipient display presented in therecipient display; and anchoring and scaling the video sequence of theextracted moving image of at least the face of the person with respectto the second background presented in the recipient display.
 2. Themethod of claim 1, wherein the recipient device includes a recipientcamera and the second background is captured by the recipient camera. 3.The method according to claim 1, wherein the extracted moving image ofat least the face of the person comprises a head and/or face of a senderof the image.
 4. The method according to claim 1, wherein the videosequence of the extracted moving image of at least the face of theperson includes one or more graphical elements.
 5. The method accordingto claim 4, wherein the one or more graphical elements is anchored tothe extracted moving image of at least the face of the person.
 6. Themethod according to claim 1, wherein the moving image of at least theface of the person is extracted from the video of the live camera viewusing subject feature detection, subject color modelling or subjectshape detection.
 7. The method according to claim 1, wherein sending thevideo sequence of the extracted moving image of at least the face of theperson to the recipient device using a messaging application executed onthe sender device.
 8. The method according to claim 1, wherein at leastthe sender device is a mobile hand held communication device.
 9. Amessaging system, comprising: a plurality of communicating devices thatincludes a sending user device and a receiving user device; wherein thesending user device comprises: a camera configured to record a video ofa live camera view including at least a face of a person and a firstbackground; and a processor configured to extract a moving image of atleast a face of the person from the video of the live camera view tocreate a video sequence, the extracted moving image of at least the faceof the person excluding images of the first background from the livecamera view, wherein the video sequence is sent as a message to thereceiving user device; wherein the receiving user device comprises: aprocessor configured to superimpose the video sequence of the extractedmoving image of at least the face of the person received from the mobilesending user device on a second background displayed on the display ofthe receiving user device, wherein the processor of the receiving userdevice anchors and scales the video sequence of the extracted movingimage of at least the face of the person with respect to the secondbackground.
 10. The system according to claim 9, wherein the receivinguser device has a camera and the second background is captured by therecipient user device camera.
 11. The system according to claim 9,wherein the video sequence of the extracted moving image of at least theface of the person is sent to the recipient device using a messagingapplication executed on the sender device.
 12. The system according toclaim 9, wherein at least the sender device is a mobile hand heldcommunication device.